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INTRODUCTION 



In response to requests received from Chapter 1 staff for 
information on how Local Education Agency (LEA) Chapter 1 staff 
might work more effectively with test publishers, the Test Infor- 
mation Center (TIC-TAC) recently conducted a telephone survey of 
selected publishers of norm-referenced tests commonly used by 
Chapter 1 LEAs. The test publishers included: CTB/McGraw-Hill , 
The Psychological Corporation, The Riverside Publishing Company, 
and Science Research Asoclates, Inc. 

The survey focused on specific services test publishers 
provide LEA staff in: selecting tests and appropriate scoring 
services; minimizing turnaround time; verifying scoring reports; 
utilizing test administration results for instructional planning; 
and troubleshooting — who to contact when test related problems 
are experienced . 

Results of the survey are reported here in a question-and- 
answer format. The specific questions, listed in the table of 
contents which follows, are grouped by topic: 

o test selection 

o test administration 

o selection, interpretation, and 
utilization of scoring reports 

o verification of scoring reports 

o coping with unexpected problems. 



The questions and answers were reviewed for accuracy by the 
test publishers included in the survey. However, because of 
potential changes, it is recommended that LEA staff confirm 
factual information concerning test publishers' policies 
and/or procedures with regional sales representatives on an 
ongoing basis. 

Terri Strand 
Coordinator 

Test Information Center 
Region 3 - Chapter 1 Technical 
Assistance Center 
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Test Selection 



1. HOW DO TEST PUBISHERS ASSIST LOCAL EDUCATIONAL AGENCIES (LEAS) 
IN TEST SELECTION? 

Publishers provide test examination kits, on-site presenta- 
tions by their regional sales representatives, and sometimes 
help in matching item objectives. Also, test publishers' 
catalo&s which are published annually include current in- 
formation about their respective tests and scoring services. 



2. IS IT ACCEPTABLE FOR LEAS USING THE NORM-REFERENCED MODEL 
CHANGE TO ANOTHER NORM-REFERENCED TEST FOR THEIR POSTTEST ON 
THE BASIS OF EQUATING STUDIES CONDUCTED BY OTHER LEAS OR STATE 
EDUCATIONAL AGENCIES (SEAS)? 

It has been reported that some test publishers' regional sales 
representatives have advised Chapter 1 LEA staff to use dif- 
ferent tests for pretesting and posttesting, based on equating 
studies conducted elsewhere. 

In an equating study, two different tests are administered to 
a sample of students in order to obtain comparable scores. 
However, because of student population differences, results 
of equating studies are considered applicable only to the 
specific LEA or SEA for which the study was conducted. Also, 
although tests are equated statistically, they may not be 
correlated in content and may vary in number of test items. 

Information about conducting an equating study is available 
from the regional Technical Assistance Centers (TACs) for 
Chapter 1 Evaluation. 



3. IF LEAS USE THE SAME NORM-REFERENCED TEST FOR PRETEST AND 

POSTTEST, WHEN CAN THEY CHANGE TO ANOTHER NORM-REFERENCED TEST 
THAT MAY BE MORE SUITABLE? 

LEAs which test in fall and spring may change tests in fall, 
at the beginning of the new evaluation cycle. 

Subject to SEA approval . Chapter 1 LEAs who test only once 
a year have several options: 
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3. (continued) 

o Double test — administer the "old" test as 
the posttest to complete the previous 
evaluation cycle and administer the "new" 
test as a pretest to begin the new evalua- 
tion cycle. 

o Administer the "new" test only and do not 
report evaluation for the year when the 
test is changed. 

o Conduct an equating study of the "old" and 
"new" tests. Information about conducting 
an equating study is available from the 
regional Technical Assistance Centers (TACs) 
for Chapter 1 Evaluation. 



Test Administration 



4. SHOULD CHAPTER 1 LEAS ADMINISTER TESTS AT OFF-PEAK PERIODS 
(LATE FALL TO EARLY SPRING) TO GET FASTER DELIVERY OF 
SCORING REPORTS, AS SOME PUBLISHERS' SALES RESPRESENTATIVES 
HAVE SUGGESTED? 

Inasmuch as national norms are being used to compare the 
achievement of Chapter 1 students, LEAs should test at about 
the same time that the national norms groups were tested. 

With the norm referenced model f LEAs should test within two 
weeks before or aftc the norming midpoint reference date or 
within 3-6 weeks before or after the midpoint reference 
date, if interpolated norms are used. 

♦ 

CTB/McGraw Hill advises that LEAs which test annually should 
use the same quarter -month for testing each year. 



5. SOME CHAPTER 1 LEAS ARE UNABLE TO TEST WITHIN TWO WEEKS OF THE 
NORMING MIDPOINT REFERENCE DATES AND WOULD LIKE TO USE INTER- 
POLATED NORMS WITHIN THE ACCEPTABLE PERIOD OF TIME FOR THE 
NORM-REFERENCED MODEL. DO ALL TEST PUBLISHES PROVIDE INTER- 
POLATED NORMS? 

CTB/McGraw-Hill, Riverside Publishing Company, and Psycholog- 
ical Corporation provide interpolated norms for many tests 
used for Chapter 1 evaluation. Also, Science Research Asso- 
ciates, Inc., will provide interpolated norms for their new 
achievement test battery — the Survey of Basic Skills as well 
as for the SRA Achievement Series which now has 1984 norms. 
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5. (continued) 

LEAs which require interpolated norms should verify prior to 
establishing test administration dates that the publisher 
provides interpolated norms for the test being used. If not, 
procedures for calculating interpolated norms are available 
from the respective state departments of education and/or 
regional TACs, as appropriate. 



Selection, Interpretation, and Utilization of 
Scoring Reports 



6. HOW DO TEST PUBISHERS ASSIST LEAS IN SELECTING SCORING 
REPORTS? 

Test publishers provide a variety of scoring reports that 
address many different information needs of districts: 
program evaluation, item and objective analyses, diagnostic 
information, instructional grouping, regrouping of students' 
spring test administration results for fall classroom place- 
ment, detailed statistical reports for administrators and 
school boards, parent-teacher conferencing, and cumulative 
record keeping. Also, scoring reports are available at 
different levels of reporting, including individual student, 
an J class, grade, building, and district summaries. 

Test; publishers provide various services and/or publications 
designed to assist LEAs in selecting appropriate reports. 
These include: 

o consultations by their regional sales 
representatives who assist LEA staff in 
defining their information neuds and 
also, make recomaendations for relevant 
scoring reports 

o scoring packages designed to simplify 
ordering procedures— each scoring package 
includes a mix of scores and reports 
emphasizing different patterns of in- 
formation needs 

o inservice workshops by their regional 
sales representatives prior to test 
administrations and after scoring 
reports are received by LEAs 

o publications about their scoring 
services and reports and/or packages 
of different sample scoring reports 
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(continued) 

o test catalogs, published annually, 
which contain current information 
about scoring services and examples 
of scoring reports. 



LEA staff should check with publishers' regional sales repre- 
sentatives concerning the availability of specific scoring 
services, reports, and types of assistance they provide LEAs. 



WHAT DO TEST PUBLISHERS RECOMMEND CONCERNING SELECTION OF 
SCORES BY CHAPTER 1 LEAS? 

Scores are generally selected at the time that LEAs order 
their scoring service reports. In general, selection of 
appropriate scores depends on hov the scores will be used as 
well as guidelines established by the SEA for submitting 
Chapter 1 evaluation data. 

The major test publishers provide a full range of scores and 
publications which explain the use of the various scores and 
their relative advantages and disadvantages. 

Frequently used scores for Chapter 1 evaluation include: 

o raw scores or number correct scores 
— used as the starting point in con- 
ducting quality control checks of eval- 
uation data and for identifying floor 
and ceiling effects) 

o scaled scores or expanded standard 
• scores — used for aggregating data, 
for functional level testing, and 
monitoring longitudinal achievement 

0 national percentile ranks — for 
comparing achievement with national 
norms 

0 normal curve equivalents (NCEs) 
— for calculating NCE gains. 



Test publishers vary in their recommendations to Chapter 1 
LEAs for selection of scores. For example, some test pub- 
lishers may not encourage LEAs to select the number correct 
(raw score) for their reports. LEAs should be aware that 
this score is essential when verifying the accuracy of the 
data in quality control checks. 
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7. (continued) 

CTB/McGraw-Hill most often recommends ordering scaled scores, 
national percentile ranks, end NCE scores. The Riverside 
Publishing Company encourages districts to select grade equiv- 
alent (GE) scores for the Iowa Tests of Basic Skills (ITBS) 
and scaled scores for the Tests of Achievement and Proficiency 
(TAP), national percentile ranks and NCE scores. However, 
LEAs are able to order scaled scores for the ITBS. 

For users of the Stanford Achievement Test or the Metropolitan 
Achievement Tests, the Psychological Corporation recommends 
using the Pupil Skills Analysis report which provides both 
norm-referenced scores (including raw scores, scaled scores, 
national percentile ranks and stanines, grade equivalents, and 
NCE scores) and objective referenced scores. For users of the 
Stanford Diagnostic Reading and Mathematics Tests, the pub- 
lisher recommends the basic service which provides norm- 
referenced scores (raw scores, scaled scores, national per- 
centile ranks and stanines, grade equivalents, and NCE scores) 
and mastery scores (Progress Indicators) for specific skills 
as well as instructional starting points. 

Science Research Associates, Inc. recommends selecting its 
scaled score (GSV), national percentile and stanine, and NCE. 
The publisher's List Report (the basic report that is fur- 
nished to all schools) automatically includes grade equivalent 
scores in addition to the above scores but provides LEAs with 
the opportunity to suppress them on their scoring reports. 

8. HOW SHOULD LEAS ORDER INTERPOLATED NORMS FROM THE TEST 
PUBLISHER'S SCORING SERVICE? 

Interpolated norms should be requested when scoring services 
are ordered. After test administration, materials shipped to 
the publisher should also indicate the test administration 
dates. 

LEAs ordering interpolated norms for CAT or CTBS should not 
request the specific quarter-month norms (no longer included 
in the publisher's catalog) but rather should enter the actual 
testing dates on the order form. It is recommended that LEAs 
verify which quarter-month norms we.e used in the scoring 
report. 



9. WHAT SERVICES DO TEST PUBLISHERS PROVIDE LEAS IN INTERPRETING 
SCORING REPORTS? 

Some publishers provide supplementary test publications which 
include detailed information for interpreting scoring reports 
and/or will provide an inservice workshop, upon LEA request. 
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10. HOW CAN LEAS RESOLVE PROBLEMS ASSOCIATED WITH TESTING WITHIN 
THE APPROPRIATE PERIODS OF TIME FOR THE NORM-REFERENCED 
EVALUATION MODEL BUT NOT BEING ABLE TO GET SCORING REPORTS 
BACK IN TIME FOR INSTRUCTIONAL PLANNING? 

According to test publishers , turnaround time (the date 
completed test booklets and answer sheets are received by 
the test publisher or scoring service and the date when 
scoring reports are shipped) generally ranges from two to 
three weeks (10-15 working days). 

Delays in receipt of scoring reports by districts are often 
related to practices followed by districts in preparing and 
shippping test administration materials. Some common causes 
of delays include: 

o order forms are not properly 
completed 

LEA staff often have to complete 
many unfamiliar order forms. The 
order forms should be completed 
carefully and verified to avoid 
any unecessary delays. Delays 
can result from omissions on the 
order forms for desired services v 
scores* and norms. 

o testing dates are not reported 

Test administration dates should 
be recorded on all documents, as 
requested by the publisher. When 
test administration dates are miss- 
ing, the publisher has to telephone 
or write to the LEA to obtain the 
information before scoring can be 
done. 

o wrong time of year of testing is 
reported 

0 wrong test form is reported 

o wrong grade is reported 

o wrong answer sheets are used 
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(continued) 

o answer sheets are not filled out 
properly 

Children should use appropriate 
No. 2 pencils for marking answer 
sheets and should not use colored 
pencils or ball-point pens. 

Children may fill in the answer 
sheets improperly or occasion*? 7.1/ 
may doodle on their answer sheets 
— sometimes over critical areas 
such as timing tracks. 

Stray marks on answer sheets can 
cause delays in processing time. 
Children's answer sheets should 
be checked and stray and extra- 
neous marks erased. 

o photocopied answer sheets are 
used 

Photocopied answer sheets cannot 
be processed by the scoring serv- 
ice and should not be used. 

o classes are mixed up when pack- 
aged and materials are not packed 
properly 

The publisher's detailed instruc- 
tions for batching materials for 
shipping she ild be carefully 
followed. ID sheets are used to 
identify the building, grade, and 
class. Delays may be created by 
ID sheets not being inserted or 
completed properly or photocopies 
being used. 
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(continued) 

0 materials are not shipped 
properly 

Delays are sometimes created by 
LEA staff sending incomplete 
shipments or not marking the 
boxes of materials sent to the 
scoring service. Boxes may be- 
come separated during shipment 
and the publisher's staff can't 
tell if the entire shipment was 
received. The publisher sug- 
gests marking boxes "1 of — , 
2 of — , etc." 

o there are delays in returning 
answer sheets 

Schools may delay shipment to 
scoring service for absentee 
children. This nay vary from 
one to two days or for weeks 
if there is a chicken pox or 
flu epidemic. 



Verification of Scoring Reports 



IS IT ADVISABLE FOR LEAS TO VERIFY THE ACCURACY OF THEIR 
SCORING REPORTS? ... IS IT NECESSARY TO VERIFY EACH STUDENT'S 
TEST SCORES? 

Errors on scoring reports may result from various factors 
relating to the ways in which the LEA submitted materials 
to the publisher (for example, time of year or year reported 
on header sheets is inaccurate, students were given the wrong 
answer sheet to complete, etc.) or in actual data processing. 

LEA staff should verify the scoring reports to ensure the 
reliability of the LEA's Chapter 1 evaluation results. In a 
small district, LEA staff may wish to verify each student's 
score, whereas in a large district, LEA staff may sample 
students' scores, for example, check every tenth score. 
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(continued) 

In verifying scoring reports, LEAs should check that: 

o scores are reported for the cor- 
rect grade level, test level, 
and test form. 

o score conversions (RS-SS, SS-?R , 
and FR-NCE) are correct for the 
correct test level and form when 
empirical norms were used (test 
was administered 2-4 weeks on 
either side of the norms midpoint 
reference date). 

o score conversions (RS-SS, SS-FR, 
and PR-NCE) are correct for inter- 
polated norms (when the test was 
administered from 3-6 weeks beyond 
the norms midpoint reference date. 

o score conversions (RS-SS) are 
correct when students in the same 
grade were administered different 
functional level tests. 



HOW CAN LEAS VERIFY THE ACCURACY OF SCORING REPORTS FOR TESTS 
ADMINISTERED FROM 3-6 WEEKS BEYOND THE MIDPOINT REFERENCE 
DATE? 

CTB/McGrav-Hil 1 provides books of interpolated norms tables 
for CAT and computer-generated interpolated norms tables 
(SS to PR or NCE) for CTBS (for both Number Correct and IRT 
scaled scores). LEAs may request the appropriate norms table 
from the publisher. There are no score conversion tables for 
Number Correct to IRT scaled scores which can only be obtained 
from the publisher's scoring service. 

The Riverside Publishing Company provides interpolated NCE 
scores as a scoring option for Gates-MacGinitie Reading Tests, 
ITBS/TAP, and The 3-R's Test. 

The Psychological Corporation routinely provides the appro- 
priate interpolated norms tables to districts which request 
interpolated norms from the publisher's scoring service. 

Science Research Associates, Inc. provides interpolated norms 
as a scoring option. Both empirical and interpolated norms 
appear on the scoring report. 
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13. HOW CAN LEAS WHICH TESTED CHAPTER 1 STUDENTS IN THE SAME 
GRADE WITH BOTH IN-LEVEL AND OUT-OF-L2VEL TESTS VERIFY THAT 
THE SCORING SERVICE USED APPROPRIATE NORMS IN CALCULATING NCE 
AVERAGES? 

LEAs should check that the publisher provides for scoring 
mixed test levels administered to students in the same grade. 
Mixed test levels for the same grade are batched together and 
during scoring, the scanner identifies the test level used by 
each student in the class and scores the answer sheet appro- 
priately. The test level administered to each student appears 
on the scoring report. 



14. WHAT PROCEDURES DO TEST PUBLISHERS USE TO DERIVE AVERAGE NCSS? 

Test publishers use different procedures for obtaining average 
NCEs and sometimes the publisher may change procedures. 

At the present time, Riverside Publishing Company and Science 
Research Associates average individual NCE scores; CTB/McGrav- 
Hill converts average scaled scores to the corresponding NCE 
scores; and Psychological Corporation converts average scaled 
scores to national percentiles and then to NCEs. 



15. DO TEST PUBLISHERS HAVE QUALITY CONTROL PROCEDURES FOR 

VERIFYING THEIR SCORE REPORTS? ... IF SO, CAN ERRORS STILL 
OCCUR ON THE SCORING REPORTS? 

The major test publishers reported using several human and 
machine quality control checks: verifying order forms and 
materials when they arrive from the district, screening 
answer sheets, verifying scanner results, and inspecting 
reports for irregularities. Most publishers reported better 
than 99 percent accuracy of scoring reports. However, scor- 
ing errors may occur even though the error rate is small. 



16. LEAS OCCASIONALLY FIND ERRORS WHEN VERIFYING THEIR SCORING 
REPORTS. ™ CAN THEY REPORT THIS TO AND HOW DO TEST 
PUBLISHERS HANDLE SUCH ERRORS? 

Test publishers suggested that if LEAs think they have iden- 
tified errors in their scoring reports, they should first con- 
tact the publisher's regional sales representative. Sometimes 
the LEA may have used an incorrect score conversion table to 
verify data. 
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16. (continued) 

CTB/McGraw-HH 1 suggests that LEA staff who have identified 
errors in their scoring reports contact Customer Services in 
Monterey (800/538-9547 or'^in CaliforniajKX3/682r9222K LEAs 
calling in will be connected will the appropriate consultant. 
The product manager or Statistical Services staff are most 
often consulted for problems relating to scoring. 

The Psychological Corporation suggested that most problems 
identified by LEA staff could be resolved by telephone. The 
publisher provides a hotline number (216/826-2879) for LEAs 
to use if they suspect that there are errbrs in their 
reports. Also, the publisher maintains a terminal in the 
Cleveland office that is used to track scoring reports. 

At other publishers' offices, LEA staff should contact 
Customer Services, advise them of the problem being exper- 
ienced, and request to speak to the person who can help them. 
The telephone number for Riverside Publishing Company is 
800/323-9540 or 312/693-0040 in Illinois, Hawaii, or Alaska, 
and for Science Research Associates, Inc., is 312/984-7000. 

Publishers CTB/McGraw-Hill , Riverside Publishing Company, 
Psychological Corporation, and Science Research Associates, 
Inc. each indicated that, in the case of errors, the pub- 
lisher would check the answer sheets and rerun the data as 
necessary. 



17. HOW LONG AFTER SCORING DO TEST PUBLISHERS KEEP COMPLETED 
TEST BOOKLETS AND SCORING SHEETS? 

Currently, CTB /McGraw-Hill keeps materials for 60 days unless 
they are requested by the LEA. The publisher keeps data on 
tape for three years. 

The Riverside Publishing Company keeps answer documents for 
three months and data tapes for 13 months. 

The Psychological Corporation keeps materials for about 13 
months. 

Science Research Associates, Inc. keeps materials for 45 days 
after reports are mailed. Thus, LEAs have 30 days after they 
receive their reportsto request return of their materials. 
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SOKE L AS WOULD LIKE TO HAVE STUDENTS' TEST BOOKLETS AND 
ANSWER SHEETS RETURNED TO THEM. DO TEST PUBLISHERS PROVIDE 
THIS SERVICE? . . . WHEN SHOULD LEAS REQUEST THIS SERVICE? 
... HOW LONG WILL IT TAKE TO RECEIVE THE MATERIALS? ... ARE 
THERE ANY CHARGES? 

CTB/McGrav-Hill, Riverside Publishing Company, Psychological 
Corporation, and Science Research Associates, Inc. indicated 
that they do provide this service. 

Districts should request this service on their order form for 
scoring reports. Publishers mail the materials by UPS or 
fourth class mail a few days after the scoring reports are 
mailed. Thus, districts will receive the materials about 3 - 
4 weeks after they receive their scoring reports. 

Districts are required to pay postage charges. Currently, 
some publishers do not charge any handling fee for the serv- 
ice while others may require a nominal amount. 



Coping with Unexpected Problems 



WHO SHOULD LEA STAFF CONTACT WHEN THEY EXPERIENCE UNEXPECTED 
PROBLEMS, SUCH AS: DELAYS OR ERRORS IN SHIPMENT, INCORRECT 
SCORING REPORTS, OR OTHER TYPES OF PROBLEMS AND ARE UNABLE TO 
CONTACT THE PUBLISHER'S REGIONAL SALES REPRESENTATIVE, OR ARE 
DISSATISFIED WITH THE REPRESENTATIVES 'S SERVICES? 

CTB/McGraw-Hill advises LEAs to call Customer Services in 
their regional office or Monterey. The publisher's staff is 
trained to answer questions, interpret test scores, and can 
determine who to contact to resolve specific problems. Cur- 
rently, the publisher provides a special answering service in 
Monterey designed to expedite LEAs' telephone requests for 
consultations concerning specific problems. 

The Psychological Corporation suggested that LEA staff con- 
tact Customer Services in the publisher's regional offices to 
resolve any problem relating to ndsshipment of materials or 
problems relating to test administration (ask to speak to the 
Administrative Assistant). District staff may also call the 
publisher's telephone hotline number to get help in complet- 
ing their orders for scoring services, obtain an explanation 
of a scoring option, ask for correction of scoring service, 
request special scoring services, check progress on their 
scoring orders, or get help in interpretation of scoring 
reports. 
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(continued) 

The Riverside Publishing Company advises LEA staff to call 
Customer Services where they will be referred to the appro- 
priate staff member. The director of marketing responds to 
questions concerning late delivery of test materials or 
dissatisfaction with the regional sales representative's 
services and the director of testing responds to questions 
relating to test content. 

Science Research Associates, Inc. advises that LEA staff 
contact Customer Services who will identify the appropriate 
staff member to consult with. The director of testing and 
his staff or the scoring services facilitator will also 
respond to a variety of questions regarding scoring services 
and interpretation of reports. 



